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ABSTRACT 



This thesis explores the dimensionality of the Defense Language Institute’s (DLI) 
primary student feed back tool - the Automated Student Questionnaire (ASQ). In 
addition a data set from ASQ 2.0 (the newest version) is analyzed for trends in student 
satisfaction across the sub-scales of sex, pay grade, and Defense Language Proficiency 
Test (DLPT) results. 

The method of principal components is used to derive initial factors. Although an 
interpretation of those factors seems plausible, they are subjected to a factor analysis 
rotation (varimax) and five factors are determined and interpreted. Each of the five 
factors are interpreted in terms of student satisfaction with DLLs: (1) academic 
environment, (2) military environment, (3) non-barracks dormitory living conditions, (4) 
official and supplemental course audio tapes, and (5) service unit ’s computer learning 
centers. From the factor loading matrix factor scores equations are developed for use in a 
sub-scale trend analysis. 

Using non-parametric procedures, each factor is checked for differences in central 
tendency by sex, pay grade, and DLPT score (DLPT consists of three tests DLPTL, 
DLPTR, DLPTS). From this analysis the following results derive: (1) sex has no effect on 
any of the factors, (2) pay grade affects satisfaction with the military environment, (3) 
DLPTL, DLPTR, and DLPTS affect satisfaction with the academic environment, and 
DLPTS also affects satisfaction with the computer learning centers. 
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I. 



INTRODUCTION 



This chapter provides some background on the Defense Language Institute (DLI) 
then addresses the problem this thesis treats and the methodology employed. 

A. BACKGROUND 

This section briefly describes DLLs mission, vision, goals, student population, 
entrance requirements, language assignments, and proficiency standards. The genesis of 
their primary student feedback tool, the Automated Student Questionnaire, (ASQ) is also 
presented. 



1. DLI — Mission, Vision, and Goals 

DLI’s home page describes their mission in the following way: “To train , sustain , 
and evaluate foreign language skills...” As the world’s “...largest...” and “...most 
respected language resource institute...” their vision for the 21st Century is to remain 
“...flexible and responsive to the changing world order and on the cutting edge of technical 
innovations...” From their goals statement, those germane to this thesis are as follows: (1) 
Training - 80% of students achieve level two proficiency in listening, reading, and 
speaking, (2) Evaluating - develop valid and reliable assessment tools, (3) Quality 
Philosophy - create a culture of continuous quality improvement, (4) Quality of Life - 
develop and maintain affordable quality of life programs that allow students to focus on 
learning. 

2. Student Population 

Each year the Defense Language Institute trains thousands of military, DOD, and 
civilian personnel in a wide variety of foreign languages. The Institute plans for 2,900 
students annually and since its inception in 1941 has graduated more than 155,000. This 
student population is screened for a minimum aptitude level prior to enrollment at DLI. 
Military students are first screened for minimum language aptitude via the Armed Services 
Vocational Aptitude Battery (ASVAB). Those that pass the ASVAB’s language aptitude 
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section are then evaluated with the Defense Language Aptitude Battery (DLAB). The 
DLAB is based on an artificial language and is designed to measure the student’s ability to 
learn foreign language in general, not his/her ability to learn a specific foreign language. 

3. Language Assignment 

Students are assigned to a specific language based on their DLAB score. Once 
assigned, the specific course of instruction varies in length and difficulty depending on the 
language. Languages are categorized on a difficulty scale ranging from one to four. A 
category one language is a close cognate of the student’s native language (assumed to be 
English) such as Spanish or French. These languages have common words, similar 
sounds, sentence structure, and recognizable symbols (letters). In contrast, category four 
languages are considered “truly” foreign languages and essentially have nothing in 
common with English. Currently 24 languages and dialects are taught and they are 
organized into seven different language schools: Asian 1, Asian 2, Middle Eastern 1, 
Middle Eastern 2, East European 1, East European 2, and West European/Latin 
American. These schools are organized into departments that employ the Team Teaching 
Concept with teams of six instructors responsible for three groups of about ten students 
each. 



4. Completion of Training 

Upon completion of training, students are evaluated for language proficiency by 
another standardized test, the Defense Language Proficiency Test (DLPT). The DLPT is 
best considered as three separate tests: (1) DLPTL, for “listening,” (2) DLPTS, for 
“speaking,” and (3) DLPTR, for “reading.” For each of the three, the scale on which 
proficiency is recorded is an eleven point scale (0, 0+, 1, 1+, 2, 2+, 3, 3+, 4, 4+, 5) on 
which zero represents no proficiency at all and five represents a proficiency level 
associated with a native speaker. As noted in the goals statement, the targeted 
achievement level for the institution is that 80 percent of the students score two or better 
on each of DLPTL, DLPTS, and DLPTR. This targeted achievement level can be roughly 
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equated to the proficiency level expected from a college language major after two full 
years of study. 

5. The ASQ 

Upon completion of the training process the students are required to provide input 
as to their perception of quality with regard to their overall DLI experience. The data 
collection instrument has traditionally been survey type forms. These surveys evolved 
from paper and pencil to their current fully automated configuration called the ASQ. 

There are two versions of the ASQ, ASQ 1.0 introduced in March of 1993, and the 
current version, ASQ 2.0, implemented in December 1995. Files with ASQ 1.0 data 
contain approximately ten thousand records while files with ASQ 2.0 data contain fewer 
records and number approximately one thousand. Data from both ASQ’s are accessible 
and can be linked to various other respondent items such as: pay grade, class number, 
service component, sex, DLAB, and DLPT score. 

The ASQ’s purpose is to collect student opinion data on the overall quality of their 
DLI experience. This task is facilitated by dividing the ASQ into two parts, each aimed at 
collecting data on one of two aspects of the students’ entire DLI experience, these aspects 
are as follows: (1) Instructional Effectiveness (IE) and (2) Program Effectiveness (PE). 

a) ASQ 1.0 

ASQ 1.0 is further divided into sub-categories within IE and PE. Each of 
these sub-categories attempts to capture student opinion on a particular dimension. These 
dimensions are designed on an intuitive basis and are not disclosed to the respondent 
during the data collection process. Each main category of IE or PE is divided into five to 
eight sub-categories, where each sub-categoiy is supported by one to nine questions. For 
example, one of the sub-categories under IE is “Counseling and Assistance” which is 
supported by three questions which are shown below: 

(1) Went out of his or her way to help me learn the Language. 

(2) Gave me helpful, individual feedback when I needed it. 

(3) Was readily approachable for counseling or assistance when I needed it. 
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b) AS Q 2.0 



The total number of questions and the number of sub-categories in both IE 
and PE are reduced in ASQ 2.0. For the new PE there are now fourteen questions that 
are believed to support only two broad sub-categories: academic and military. The new 
IE arrangement is similarly transformed. 

6. Other Feedback Sources 

DLI staffs internal evaluation teams in the form of Training Assistance Visits 
(TAV) and hosts external committees to aid them in the curriculum review process. 

TAVs and curriculum reviews are expected to provide information leading to better 
understanding of root causes of problems experienced by students. TAV reports are 
qualitative descriptions of the school’s management practices, procedures, and their 
impact on the training program. 

Curriculum reviews were conducted only up to the fall of 1993; however, staffing 
and procedures for re-establishing a revised version of the curriculum review are now 
under discussion. The primary function of the curriculum review was to furnish the 
customer agencies that use DLI graduates a first hand look and direct input into the 
language programs. 

B. PROBLEM STATEMENT 

In accordance with the Institute’s goals of “continuous quality improvement” and 
maintenance of “valid and reliable assessment tools,” DLI staff is evaluating student 
feedback as an indicator of Program and Instructor Effectiveness. They want to know if 
their ASQ really measures the underlying phenomena it purports to measure. Specifically, 
with respect to ASQ 2.0: 

(1) Do ASQ 2.0’s 14 questions support the two sub-category design? 

(2) If not, what is ASQ 2.0’s dimensionality and which questions represent those 
dimensions? 
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Additionally, DLI’s Research and Analysis Division would also like to see some 
analysis of the ASQ 2.0 responses to determine if student satisfaction levels appear to 
depend on pay grade, sex, or DLPT scores. 

C. METHODOLOGY 

First the ASQ 2.0 data are analyzed to determine the number of separate 
underlying mathematical factors that account for the variability in the data set, in other 
words, to determine the mathematical dimensionality. This is accomplished using the 
method of principal components. Once the principal components are determined, a factor 
analysis is performed in an attempt to improve the interpretations of the components or 
factors. The number of factors are then compared to the number of dimensions believed 
to be designed into the questionnaire. The dimensional design was not disclosed to the 
author until after the factor analysis was completed and interpretations made. The 
mathematical factors are qualitatively analyzed to examine whether they make intuitive 
sense as well. 
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D. DIMENSIONALITY ANALYSIS 



This chapter presents an analysis of the ASQ 2.0 PE data. The 14 
questions (variables) are first analyzed for pairwise correlation followed by 
principal components analysis. Several methods for eliminating components are 
discussed; then, using a composite of those methods seven of the fourteen 
principal components are eliminated and five of the remaining seven are 
interpreted. Various numbers of components are retained and subjected to the 
varimax factor analysis rotation scheme and subsequent interpretation of the 
rotated factors is presented. The varimax factor analysis rotation maximizes the 
sum of the squared correlations for retained components thereby producing a 
unique rotation matrix which, when multiplied by the original matrix of 
eigenvectors, produces a unique factor matrix. 

A. THE DATA 

As stated in the introduction, the variables of this data set are the responses 
to questions concerning program effectiveness at DLI, and the respondents are 
DLI students. 



1. The Variables 

There are fourteen variables and the response to each is an integer in the 
range 0-4. The responses represent the students’ opinions on various aspects of 
their entire DLI experience. For each question the possible responses are: 

(1) “0” No opinion or not applicable 

(2) “1” Strongly Disagree 

(3) “2” Disagree 

(4) “3” Agree 

(5) “4” Strongly agree 

All questions are worded so that the higher number response relates to the 
question in a positive way, e.g., “The course objectives were clearly explained to 
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me early in the program.” rather than, “The course objectives were not well 
explained .” Therefore, for all questions a higher number is interpreted as better 
than a lower number. The entire set of questions is shown in Figure 1. 



1. The overall instructional program for my language was well organized. 

2. The overall program goals and related requirements were clearly explained. 

3. Teaching team cooperation fostered an effective learning environment. 

4. The grading system was clearly explained early in the program. 

5. The official course tapes had good sound quality. 

6. The official text materials were useful. 

7. Course tests appropriately measured my ability. 

8. Supplemental (instructor-prepared) language tapes had good sound quality. 

9. My service unit's computer learning center was helpful in supporting my study 
program. 

10. My barracks/dormitory was comfortable and well maintained. 

11. My government housing was comfortable and well maintained. 

12 The quality of the food in the dining facilities was good. 

13. My military training at DU contributed to my overall development. 

14. The care 1 received at the Troop Medical Clinic was responsive to my needs. 



Figure 1: ASQ 2.0 Questions 



2. The Respondents 

The set of respondents for this study consists of 615 DLI students. All 
students are unique in that each contributes only one set of responses to the data 
base. The students in the data base are summarized by pay grade and sex in Figure 
2 The vertical axis contains pay grade-sex combinations and the horizontal axis is 
the number of respondents. 
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Figure 2: Number of Respondents by Subsample 
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B. VARIABLE CORRELATIONS 



Correlations between pairs of variables are computed from the 615 
observations on each question and are shown below in Table 1. Combinations that 
appear to be “reasonably” correlated are shown in boldface type. 





Ql 


Q2 


Q3 


Q4 


Q5 


Q6 


Q7 


Q8 


Q9 


Q10 


Qll 


Q12 


Q13 


Q14 


Ql 


1.00 


.530 


.425 


332 


.251 


353 


371 


.186 


.105 


.072 


.025 


.139 


.167 


.085 


Q2 




1.00 


363 


352 


.185 


309 


.291 


.083 


.133 


.079 


.069 


.126 


.170 


.088 


Q3 






1.00 


.265 


.137 


.233 


.273 


.175 


.089 


.035 


.027 


.006 


.045 


.079 


Q4 








1.00 


.154 


.240 


.269 


.094 


.024 


.051 


.004 


.066 


.089 


.073 


Q5 










1.00 


.185 


.183 


396 


.119 


-.06 


.063 


.036 


.104 


.067 


Q6 












1.00 


303 


.146 


.049 


.081 


.055 


.123 


.118 


.139 


Q7 














1.00 


.096 


.059 


-.02 


-.04 


.037 


.084 


.088 


Q8 
















1.00 


.017 


.046 


.071 


.061 


.030 


.126 


Q9 


















1.00 


.260 


-.04 


.172 


.202 


.087 


Q10 




















1.00 


-.34 


397 


.325 


.181 


Qll 






















1.00 


-.005 


.009 


.039 


Q12 
























1.00 


342 


.220 


Q13 


























1.00 


.286 


Q14 




























1.00 



Table 1: New ASQ correlation coefficients 



From this correlation matrix several possible groupings of variables are 
readily apparent. Most notable is the group that contains Ql, Q2, Q3, Q4, Q6, 
and Q7. Two other groups that are less well correlated are Q5 with Q8, and the 
group consisting of Q 10, Q 12, and Q13 

C. PRINCIPAL COMPONENTS 

Principal component analysis is a technique for transforming a set of 
original, interrelated variables into a set of new uncorrelated variables where each 
component is a linear combination of the original variables and the “information” 
contributed by each component is proportional to its variance (Afifi and Clark 

1 990). Thus for N original variables x / , JC 2 Xn we want to arrive at a set of 

new variables C/ , C2,..., Cn where: 
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Cj=a lj x 1 + a 2j x 2 +...+ a ln x N 



subject to the following three constraints: 

(1) The variance of Cj > variance C 2 ^ ... > variance Cm 

(2) All Cj are orthogonal 

(3) For all C j, d jj+ a 2 2 /+...+ a 2 jm~ 1 

Prior to performing the principal components analysis the original variables 
are standardized (each is divided by its standard deviation) so that the principal 
components are found for the correlation matrix rather than the covariance matrix. 
According to Afifi and Clark (1990) this simplifies the analysis because: (1) The 
total variance of the system is now equal to the number of variables N, ( 2 ) the 
correlation between the i th principal component and the j th variable is simply 
Vj—aij VvAJrTc / , where VAR C, is the eigenvalue of the i th column. From the 
correlation matrix shown above as Table 1 the principal components of the data 
are calculated and shown below in Table 2. 

In Table 2 each column is an eigenvector and the eigenvalue shown 
represents the variation accounted for by that eigenvector (each eigenvector is a 
principal component). The principal components run sequentially (one through 
fourteen) from left to right in descending order based on the magnitude of their 
eigenvalue. The row labeled “Percent” shows the percent of variation accounted 
for by that particular eigenvector. For example, in column one the eigenvalue, or 
total variation accounted for by the first principal component (PC-1) is 3.09. The 
total variation is equal to the number of variables, in this case 14, making both the 
individual and cumulative percentage of variation accounted for by PC-1 equal to 
3.09/14 which we can see from Table 2 is 22.08 percent (approximately, due to 
rounding). Similarly in Table 2, the row labeled “CumPercent” shows the 
percentage of total variation accounted for by the summation of eigenvalues from 
column one through column fourteen. To calculate the cumulative percentage for 
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any column, the eigenvalue of that column and all previous eigenvalues are 
summed and the sum is divided by 14. 

Following Afifi and Clark (1990), variables that have a correlation 
coefficient greater than .5 with a principal component are considered highly 
correlated and said to “load” on that component. For the table of eigenvectors 

shown below this amounts to values of ay > .5/ VVAR C, (where C ,• is column /' 
from Table 2, and ay is the entry at the intersection of the i th column and f h row); 
these values are shown in boldface type in the body of the table. Again, several 
groupings of component-variable combinations are apparent. Based on Afifi and 
Clark’s criterion we see that variables Ql, Q2, Q3, Q4, Q6, and Q7 load on PC-1. 
This grouping is identical to the most prominent grouping surfacing in the analysis 
of the correlation matrix of Table 1. In similar fashion, variables Q10, Q12, and 
Q13 load on PC-2, which also parallels the correlation matrix findings. 





PC-1 


PC-2 


PC-3 


PC-4 


PC-5 


PC-6 


PC-7 


PC-8 


PC-9 


PC-10 


PC-11 


PC-12 


PC-13 


PC-14 


Eigenvalue : 


3.09 


1.88 


1.23 


1.06 


0.95 


0.83 


0.81 


0.76 


0.70 


0.66 


0.60 


0.57 


0.44 


0.41 


Percent : 


22.08 


13.41 


8.81 


7.55 


6.80 


5.94 


5.76 


5.44 


5.00 


4.71 


4.30 


4.07 


3.16 


2.95 


CumPercent : 


22.08 


35.49 


44.30 


51.85 


58.65 


64.60 


70.35 


75.80 


80.80 


85.51 


89.81 


93.89 


97.05 


100.0 


Eigenvectors : 






























Ql 


0.42 


- 0.15 


- 0.12 


0.04 


0.07 


0.13 


- 0.07 


0.07 


- 0.29 


- 0.09 


- 0.14 


- 0.22 


- 0.71 


0.30 


Q2 


0.39 


- 0.12 


- 0.15 


0.22 


0.19 


0.17 


- 0.10 


- 0.06 


- 0.11 


- 0.24 


- 0.27 


- 0.47 


0.53 


- 0.22 


Q3 


0.32 


- 0.19 


- 0.16 


- 0.03 


0.06 


0.46 


0.43 


0.06 


- 0.16 


0.00 


0.11 


0.60 


0.13 


- 0.10 


Q4 


0.31 


- 0.14 


- 0.19 


0.12 


- 0.13 


0.05 


- 0.24 


- 0.62 


0.55 


0.16 


0.13 


0.10 


- 0.10 


- 0.01 


Q5 


0.24 


- 0.14 


0.35 


- 0.46 


0.19 


- 0.35 


- 0.24 


- 0.25 


- 0.14 


- 0.25 


- 0.26 


0.34 


0.14 


0.15 


Q6 


0.33 


- 0.09 


0.01 


0.08 


- 0.23 


- 0.26 


- 0.15 


0.60 


0.41 


- 0.38 


0.18 


0.13 


- 0.02 


- 0.12 


Q7 


0.31 


- 0.18 


- 0.19 


- 0.01 


- 0.14 


- 0.53 


0.18 


0.11 


- 0.21 


0.61 


0.08 


- 0.08 


0.21 


0.14 


Q8 


0.19 


- 0.08 


0.42 


- 0.57 


- 0.22 


0.37 


- 0.02 


0.09 


0.07 


0.23 


0.25 


- 0.36 


0.02 


- 0.17 


Q9 


0.16 


0.28 


- 0.01 


- 0.17 


0.73 


- 0.13 


0.32 


0.09 


0.38 


0.13 


0.01 


- 0.10 


- 0.14 


- 0.12 


Q10 


0.15 


0.54 


- 0.22 


- 0.18 


- 0.09 


0.20 


- 0.09 


0.08 


0.09 


- 0.06 


0.08 


- 0.01 


0.28 


0.67 


Ql 1 


0.01 


- 0.25 


0.58 


0.49 


0.25 


0.16 


- 0.05 


0.11 


0.12 


0.18 


0.12 


0.03 


0.13 


0.43 


Q12 


0.20 


0.43 


0.15 


0.16 


- 0.05 


0.13 


- 0.41 


0.18 


- 0.07 


0.42 


- 0.40 


0.27 


- 0.06 


- 0.30 


Ql 3 


0.23 


0.39 


0.20 


0.22 


0.05 


- 0.16 


- 0.05 


- 0.24 


- 0.38 


- 0.17 


0.63 


0.00 


- 0.04 


- 0.19 


Q14 


0.18 


0.26 


0.35 


0.16 


- 0.41 


- 0.11 


0.60 


- 0.19 


0.14 


- 0.14 


- 0.36 


- 0.08 


- 0.06 


0.02 



Table 2: Principal components 



Also of interest is that Q14 is represented by PC-7. Q5 and Q8, which 
appeared correlated earlier, do not pass Afifi and Clark’s loading criterion on any 
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component, however Q5 correlates with PC-1 at the revalue of .41, and Q8 with 
PC-3 at an of .46, both just slightly below the cutoff level. 

1. Component Consistency 

Before performing any further analysis on the principal components, 
verification that the apparent structure really exists must be obtained. If the 
components really exist then they should also be apparent when the data are 
divided into smaller or arbitrary sub-sets. To check the consistency of the 
underlying structure the data are divided using three separate schemes: (1) time, 

(2) number of records, and (3) random sort (each will be described in a separate 
paragraph to follow). To obtain “pictures” of the data structure GHP biplots are 
produced for the data reference set and sub-sets. Biplots project higher 
dimensional spaces onto lower dimensional sub-spaces: in our case the fourteen 
dimensional data space is projected onto a three dimensional sub-space. GH' 
biplots scale the principal components so that the standard deviation of each is 
equal to one. They are better at illustrating the variable correlations than are the 
standard or “JK” biplots. The rays numbered one through fourteen represent the 
original variables and clusters of rays indicate groups of variables that are highly 
correlated. All the GH' biplots shown below give views from the same perspective 
for ease of comparison. 

a) The Reference Picture 

The components and rays derived from the entire data set are used 
to produce the reference picture given by the GH 1 biplot in Figure 3. The 
variables’ three major groupings are circled and labeled in Figure 3 below. In the 
following comparisons the sub-set biplots are compared to this original picture 
with respect to cluster composition and orientation on the X, Y, and Z axes. In all 
cases the clusters orient as follows: (1) group one points down and to the right and 
consists of Ql, Q2, Q3, Q4, Q6, and Q7, (2) group two points down and to the 
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left and consists of Q5, Q8, and Q1 1, (3) group three points up in the direction of 
the Y axis and consists of Q9, Q10, Q12, Q13, and Q14. 




b) Division by time 

For a first look at an arbitrary division of the data, the median ASQ 
completion date is chosen as the dividing line. This division puts a total of 223 
records in sub-set one, and the remaining 392 records in sub-set two. GH' Biplots 
of the two sub-sets are shown below in Figure 4 with sub-set one as Figure 4a and 
sub-set two as Figure 4b. It is apparent that both divisions of the data have 
roughly the same shape as the reference biplot. In Figure 4a we see the same three 
groupings of rays but with slightly more “distance’' among rays within groups as 
compared to Figure 4b; this greater variability may be caused by the smaller 
number of records allocated to sub-set one. 
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Figure 4: GH 1 Biplots of data divided by time 



c) Dividing data set in half 

The second arbitrary grouping scheme divides the original, 
unordered data in half (approximately) with 307 records in sub-set one (Figure 5a) 
and 308 records in sub-set two (Figure 5b). Again, GH' biplots are presented 
below and we see the familiar shape of the reference data set reflected in both 
Figure 5 a and Figure 5b lending more credence to the existence of a consistent 
underlying structure. 
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Figure 5: GH' Biplots of Data Divided in Half 



d) Random Division 

As a final check for consistency, the “date ordered” data are 
randomly separated into two groups. This separation was effected using the Excel 
RAND function. The division scheme is performed several times with a set of 
biplots created and analyzed for each distinct data division. All divisions evidenced 
similar shapes therefore only one set of biplots is presented below in Figure 6. 
Again, the familiar shape of the reference data set is readily apparent in the 
randomly divided data. 
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Figure 6: GH' Biplots of data divided randomly 
e) Conclusions on Consistency 

With three separate data division schemes yielding very similar 
results, the data’s underlying structure is determined to be consistent and further 
analysis is justified. 

2. Component Retention 

One objective of principal component analysis is reduction of 
dimensionality. If reduction is to be realized certain components must be culled. 
Many criteria for rejecting (or keeping) components exist and the ones selected for 
this analysis result from work done by Kaiser(1960), Cattell(1966), Afifi and 
Clark(1990) Jolliffe(1972), and Dunteman (1989). Kaiser (1960) recommends, for 
correlation matrices, dropping components with eigenvalues less than one. Jolliffe 
(1972) claims, based on simulation studies, that Kaiser’s “less than one” criteria is 
too restrictive and often throws away much information and that for a correlation 
matrix .7 is a better target value for eigenvalues. Cattell (1966) recommends a 
“scree” graph. In the scree graph, eigenvalues are plotted in decreasing order and 
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connected by lines A point “k” is chosen where the connecting lines are “steep” 
to the left of k and “not steep” to the right of k; finally k principal components are 
selected. Afifi and Clark (1990) as well as Dunteman (1989) recommend retaining 
enough components to account for a significant cumulative percentage of 
variation; in the examples shown they used 80 percent as a guide. 

Referring to Table 2, the above criteria pose several plausible 
combinations, all yielding similar results. Following Jolliffe (1972), Afifi and Clark 
(1990), and Dunteman (1989) we select nine principal components as the ninth 
eigenvalue is .7006 and the cumulative variation accounted for by nine 
eigenvectors is 80.8034 (under PC-9 in Table 2). Selecting according to Kaiser 
(1960) we choose only four principal components where the fourth eigenvalue is 
1.06, (the fifth is .95) and the cumulative variation accounted for by four 
eigenvectors is 51.8535 (under PC-4 in Table 2). Cattell’s scree technique results 
in the graph shown in Figure 7. 




Figure 7: Scree Graph of Eigenvalues 



The slope of the scree graph decreases sharply between principal 
components 1 and 3, and then less sharply between 3 and 6 and appears to 
decrease at a constant rate through the fourteenth component. Based on the scree 
analysis six components seem to be a reasonable choice. The eigenvalue and 
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cumulative percentage of variance for six components are .8318 and 64.59 
respectively (under PC-6 in Table 2). Using the selection methods discussed 
above, a range of four to nine principal components is selected for possible 
retention. Given this range seven components are chosen for the following 
reasons: 

a) Eigenvalue and Cumulative Variation 

The eigenvalue and cumulative variation for the seventh component 
(PC-7 in Table 2) are .8062 and 70.3547 respectively; therefore, its eigenvalue is 
well within the range between Kaiser’s 1.00 and JollifFe’s 0.7 and together with the 
first six components accounts for over 70 percent of the variation. 

b) Interpretability 

Since the choice of criterion for component retention is somewhat 
arbitrary, it seems reasonable to select, from within the aforementioned range of 
four through nine components, the number of components that seems to yield the 
best interpretation. Again that number is seven and the interpretation of those 
seven components follows. 

3. Interpretation of Retained Components 

Seven principal components are retained for the reasons stated above. In 
the context of the original variables the following interpretations are presented: 

a) Component One 

An excerpt from Table 2 (shown below as Table 3) demonstrates 
that variables Ql, Q2, Q3, Q4, Q6, and Q7 are all highly correlated (o,j > 
.5/VVARC, ) with PC-1. 
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Eigenvalue: 


3.09 


Percent: 


22.08 


CumPercent: 


22.08 


Eigenvectors: 




Q1 


0.42 


Q2 


0.39 


Q3 


0.32 


Q4 


0.31 


Q5 


0.24 


Q6 


0.33 


Q7 


0.31 


Q8 


0.19 


Q9 


0.16 


Q10 


0.15 


Q11 


0.01 


Q12 


0.20 


Q13 


0.23 


Q14 


0.18 



Table 3: Excerpt from Table 2 



From Figure 1 we see that the questions listed above all have to do 
with the academic portion of the students’ DLI experience. Q1 and Q2 concern 
the overall instructional program’s organization, goals, and related requirements. 
Q4 and Q7 are about the grading system and the tests. Q3 is about teaching team 
cooperation and Q6 about official text materials. Therefore PC-1 is interpreted as 
a measure of satisfaction with students ’ academic environment. 

b) Component Two 

Again, in Table 4 we see that Q10, Q12, and Q 13 are highly 
correlated with PC-2. 



20 



Eigenvalue: 


1.88 


Percent: 


13.41 


CumPercent: 


35.49 


Eigenvectors: 




Q1 


-0.15 


Q2 


- 0.12 


Q3 


-0.19 


Q4 


-0.14 


Q5 


-0.14 


Q6 


-0.09 


Q7 


-0.18 


Q8 


-0.08 


Q9 


0.28 


Q10 


0.54 


Q11 


-0.25 


Q12 


0.43 


Q13 


0.39 


Q14 


0.26 



Table 4: Excerpt from Table 2 



As seen in Figure 1 the questions that load on PC-2 are military 
aspects of a student’s DLI experience. Q10 deals with barracks living conditions, 
Q12 with food in the dining facility, and Q13 with military training while at DLI. 
Therefore PC-2 is interpreted to be a measure of satisfaction with students ’ 
military environment. 

c) Component Three 

PC-3 appears to be highly correlated with only one question. 
However, that question lends itself to a unique interpretation in that it relates to 
government housing as opposed to barracks or dormitories. Since only one 
variable loads heavily on PC-3 no excerpt from Table 2 is shown here, however 
the variable’s a,y score is .5804. Since only this question is highly correlated to 
PC-3 and the question concerns a somewhat unique aspect applicable to a smaller 
percentage of students, PC-3 is interpreted to be a measure of satisfaction with 
students ’ non-barracks/dormitory government quarters. 
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d) Components Four and Six 



PC-4 and PC-6 have no variables that load on them at the > 

.5/ -7 VAR C, level that have not already been used for a component with a higher 
overall variance. For instance, Q1 1 loads on PC-4 at just about the .5 level; 
however, Q1 1 has already been allocated to PC-3. PC-4 does have weak negative 
correlations with Q5 and Q8 which appeared to be somewhat correlated in the 
original variables correlation matrix shown in Table 1. This weak negative 
relationship may play a part in the factor analysis phase where it will be seen that 
Q5 and Q8 appear to form their own rotated component with an interpretation 
concerning audio tapes. 

e) Components Five and Seven 

PC-5 has one variable that is highly correlated: Q9. Q9 deals with 
the respondent’s service unit’s computer learning center. Similarly, Q14 singularly 
loads on PC-7. Q14 deals with the care the respondent received at the Troop 
Medical Clinic. Therefore, component five is interpreted as a measure of 
satisfaction with students ’ service unit’s computer learning center and seven is 
interpreted a measure of satisfaction with healthcare. 

D. FACTOR ANALYSIS 

The purpose of factor analysis is to find new factors that are easier to 
interpret in the context of the original variables (Afifi and Clark 1990). Dunteman 
(1989) points out that if a suitable interpretation can be made from the original 
principal components then there is no need to rotate them. Nevertheless, for 
possible enhancements to our interpretation a factor analysis is performed. For our 
initial factors we restrict attention to the same four through nine principal 
components chosen in the previous section. Various numbers of these principal 
components are subjected to the varimax rotation scheme to see if factor rotation 
contributes anything to our understanding of the data’s underlying factor structure. 
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The varimax rotation maximizes the sum of the squared factor loadings of the 
retained components, producing a unique rotation matrix that when multiplied by 
the original matrix of eigenvectors produces a unique factor matrix. The objective 
is to produce factors that have some loadings near one and some near zero, 
thereby making them easier to interpret. The criteria for a successful factor 
analysis is taken from Johnson and Wiehert (1982) and is known as the “Wow” 
criteria: 

If, while scrutinising the factor analysis, the investigator can 
shout ‘Wow, I understand these factors!’, the application is deemed 
successful. 

As is seen in the following interpretation the “Wow” criteria is satisfied and the 
rotation adds to the understanding of the data’s structure. 

1. Interpretation of Rotated Components 

The varimax rotation scheme is applied to the candidate range of 
components four through nine. A discussion and interpretation of each is 
presented below followed by a conclusion concerning the rotation process. 

a) Rotating Four Components 

When four components are rotated the rotation matrix groups the 
questions into four distinct groups that differ only slightly from the original 
principal component structure. As with the original principal components analysis, 
Ql, Q2, Q3, Q4, Q6, and Q7 are grouped together in the rotated component one 
(RC-1). The interpretation of this component remains the same as in the principal 
component interpretation: a measure of satisfaction with students ’ academic 
environment. 

RC-2 contains all of the questions contained in PC-2 with the 
addition of Q14. This seems to make sense as Q14 pertains to adequacy of care 
received at the Troop Medical Clinic. PC-2 was originally interpreted: a measure 
of satisfaction with a students’ military environment. The Troop Medical Clinic 
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is a military activity and therefore its addition to RC-2 serves to strengthen its 
original interpretation. Therefore RC-2’s interpretation is the same as PC-2’s 
interpretation given above. 

RC-3 also follows the original principal component interpretation: a 
measure of satisfaction with students ’ non-barracks/dormitory government 
quarters. Q1 1, which deals exclusively with government housing, is highly 
correlated with this rotated component. 

Unlike PC-4, for which no clear interpretation was discernible, RC- 
4 is highly correlated with Q5 and Q8. Q5 and Q8 both deal with course (and 
supplementary, instructor prepared) tapes’ audio quality: additionally, these are the 
only questions that deal with course tapes. Therefore grouping these two 
questions together seems to add to our understanding of the data’s factor 
structure; RC-4 is interpreted as a measure of students ’ satisfaction with the audio 
quality of the official and supplemental course tapes. 

Rotating four components suggests four convenient groupings of 
the variables but it should be mentioned that the four component rotation scheme 
leaves out one variable. Q9, concerning the students’ service unit’s computer 
learning center, is not strongly correlated with any of the four rotated components. 

b) Rotating five Components 

When five original components are rotated, the first four rotated 
components are loaded on by the same variables as in the four component scheme 
discussed above, and the magnitude of the loadings do not change appreciably. 

The major change resulting from the additional rotated component is that Q9, left 
out in the four component scheme, is now included and turns out to be highly 
correlated with RC-5. Therefore, the interpretation of the first four rotated 
components remains the same as in the four component scheme and RC-5 is 
interpreted to be a measure of students ’ satisfaction with their service unit’s 
computer learning center. 
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c) Rotating Six through Nine Components 



The effect of adding a sixth component for rotation is to separate 
two variables from RC-1 . Q6 and Q7 are highly correlated with RC-6. Q6 is as 
follows: “The official text materials were useful.” Q7 reads “Course tests 
appropriately measured my abilities.” The author can see no inherent similarity in 
these two questions. The usefulness of text materials and measurement abilities of 
tests do not seem to be intuitively related. Grouping these two questions together 
- separate from the other questions concerning the overall academic environment - 
seems to add an air of ambiguity to the notion of an underlying structure. Similar 
results occur from the addition of more rotated components. A seventh 
component strips Q14 from RC-2 and isolates the variable as RC-7. This reverses 
the improvement in RC-2 over PC-2. Adding an eighth rotated component 
captures Q4 as that eighth component, further detracting from the integrity of RC- 
1. Finally, a ninth component sets apart the grouping of Q5 and Q8. 

2. Discussion 

The factor analysis indicates that ASQ 2.0 is essentially five dimensional as 
compared to the two dimensions it is designed to cover. Those five dimensions 
are: 

(1) RC- 1 : a measure of satisfaction with students ’ academic 

environment. 

(2) RC-2: a measure of satisfaction with students ’ military 

environment 

(3) RC-3 : a measure of satisfaction with students ’ non- 

barracks/dormitory government quarters. 

(4) RC-4: a measure of students ’ satisfaction with the audio quality of 

the official and supplemental course tapes. 

(5) RC-5: a measure of students ’ satisfaction with their service unit’s 

computer learning center 
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For each of the five factors selected, quantitative measures are developed 
for use in further analysis. These quantitative measures are called “factor scores.” 
Afifi and Clark(1990) point out that “it is conceivable to construct factor score 
equations in an infinite number of ways.” The factor scores chosen here are the 
original variable magnitudes scaled by the factor loading coefficient of that variable 
on that factor. This allows the variables with the highest loadings to contribute a 
greater percentage to the total factor score. The factor score equations are given 
below where factor one is labeled “RC-1” and the remainder accordingly: 

(1) RC-1 = .75334901 + .72438402 + .63164702 + .62639104 + 

.55760705 +.62927107 

(2) RC-2 = .53546015 + .70461012 + .70036012 + .67486014 

(3) RC-3 = .914917011 

(4) RC-4 = .7123805 + .83231502 

(5) RC-5 = .84917209 
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HI. FACTOR CENTRAL TENDENCY ANALYSIS 



This chapter presents analysis investigating the effects of sex, military rank, 
and DLPT score on student satisfaction. DLI’s Research and Analysis Division is 
interested in trend analysis across various sub-scales; specifically, questions 
whether student sex, military rank, or DLPT score affect student satisfaction 
levels are considered here. The rotated component scores developed in Chapter II 
are used as quantitative measures of student satisfaction. 

A. NON-PARAMETRIC METHODOLOGY 

Effects of sex, pay grade, and DLPT score on student satisfaction are 
explored using factor scores as response variables. Some of the groupings have 
small numbers of observations associated with them, making accurate 
distributional determinations unlikely; therefore, non-parametric comparisons of 
central tendency are employed, avoiding the normality assumption necessary for 
standard techniques. The data generation mechanism (ASQ’s taken from seven 
different schools with small groups of students taught by various teaching teams of 
six instructors that are not deliberately kept intact from one class to the next) 
provides a defensible guard against serial and auto-correlation; however, the 
random sampling hypothesis is untenable and this analysis suffers to the extent that 
it is violated. 

1. Tests Employed 

The object of this analysis is to determine if a factor’s central tendencies 
are the same across all groups. Mean (and median) scores for all five rotated 
components are tested using three different non-parametric tests: (1) Wilcoxon 
Rank Sum Test (Kruskal-Wallis with more than two groups), (2) Median Test 
(number of points above median), and (3) Van der Waerden Test (normal 
Quantiles). In addition, the data are tested for homoscedasticity using four 
different tests: (1) O’Brien, (2) Brown-Forsythe, (3) Levene, and (4) Bartlett. For 
handling cases of heteroscedasticity the Welch Anova is employed. 
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2 . 



Rotated Component Scores of Zero 



In all cases, rotated component scores of zero are present. The extent to 
which they appear and the effect they have on the analysis varies by question. 
Table 5 depicts the incidence rates of “0” for all five rotated components. 



ROTATED COMPONENT SCORES OF ZERO 


COMPONENT 


NUMBER OF 


NUMBER OF 


PERCENT 




ZEROS 


RESPONSES 


OF ZEROS 


RC-1 


3 


615 . 


004878 


RC-2 


33 


615 


053659 


RC-3 


391 


615 


.635772 . 


RC-4 


n 


615 


017886 


RC=5 


25£ 


£15 


41626 



Table 5: Incidence of Zero Scores for Rotated Components 
a) RC-1, RC-2 and RC-4 

RC-1, RC-2, and RC-4 all have six percent or fewer zeros. 



indicating that the majority of students answer the questions associated with those 
factors. Nevertheless, to get a zero score for any of RC-1, RC-2 or RC-4 a 
student has to respond “No opinion or not applicable” to at least two questions, 
and in the case of RC-1 that response must be provided for six questions. The 
notion that a student in a military-academic environment either has “no opinion” 
about the school or the military, or that the questions concerning the functioning of 
that environment are “not applicable” to the student seems implausible, and it is 
possible students select zero for some other unknown reason. It is noted here that 
with the high positive response rates for these rotated components the inclusion, or 
exclusion, of the zeros matters little, and since their inclusion is conservative in 
nature they are kept in for this analysis. 

b) RC-3 and RC-5 

RC-3 and RC-5, shown in Table 5, have significant incidence rates 
for zero responses. This is to be expected because RC-3 and RC-5 are both driven 
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by one question each. RC-3 is uniquely associated with Q1 1, which concerns only 
non-barracks/dormitory government housing, and RC-5 concerns the students’ 
service unit’s computer learning center. Since these questions do not pertain to all 
students the notion of “not applicable” seems plausible here; therefore, the zero 
responses are eliminated. 

B. EFFECTS OF DLPT SCORE ON STUDENT SATISFACTION 

In all three DLPT cases, DLPT score means are found to be significantly 
different with respect to RC-1 (academic satisfaction) and in the case of DLPTS a 
significant difference exists across RC-5. A graphical depiction of central 
tendencies for DLPTL, DLPTR, and DLPTS across RC-1 is shown below as 
Figure 8. 

In Figure 8 quantile boxes are shown to illustrate the response variables’ 
spread and medians; additionally, the jagged lines illustrate the means and are 
connected to highlight the upward trend. The straight horizontal line depicts the 
overall mean. 
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16 26 29 32 34 36 38 40 42 44 46 48 50 52 54 56 58 

DLPTL 




DLPTR 




DLPTS 



Figure 8: RC-1 vs. DLPTL, DLPTR, and DLPTS 
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Looking at Figure 8 above it appears that there exists an upward trend 
across all three DLPT’s indicating that higher scoring students tend to be more 
satisfied with DLLs academic environment The test results for RC-1, across the 
three DLPT’s, are presented below in Table 6. The p - values shown in the table 
body indicate the probability of obtaining, by random chance alone, a % value 
greater than the one calculated, if in fact the distributions across the factor level all 
have the same center. P-values below the .05 level are usually considered ample 
evidence that the distributions do not have the same center. With the exception of 
the Median test for DLPTL, all values are well below the .05 level. In all three 
cases, the DLPT’s heteroscedasticity results indicate that the variances are the 
same across all levels. 





RC-1 




DLPTL 


DLPTR 


DLPTS 


Kruskal-Wallis 


.0014 


.0007 


<0001 


Median 


.1689 


.0174 


.0003 


Van der Waerden 


.0009 


.0012 


<0001 



Table 6: Non-parametric Results for DLPT's vs. RC-1 



In addition to the significant finding for RC-1, it appears that students who 
score higher on DLPTS tend to have a higher opinion of their service unit’s 
computer learning center. Test results, verifying the difference in means for 
DLPTS vs. RC-5, are presented below in Table 7,. There was no indication of 
heteroscedasticity therefore the Welch Anova was not calculated. 
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Tab: 



TEST 


RC-5 


Kruskal- Wallis 


.0019 


Median 


.0046 


Van der Waerden 


.0018 



e 7: Test Results for DLPTS vs. RC-5 



Although the test results shown in Table 7 support a difference in means, 
small numbers of observations in the groups making up the “tails” of the sub-scale 
DLPTS, make the notion of a trend somewhat questionable. 

C. EFFECTS OF PAY GRADE ON STUDENT SATISFACTION 

It is noted here that military rank is not the only possible treatment level 
since data from GS employees are also available in the data base. However, RC-2, 
RC-3, and RC-5 all have direct military connotations: RC-2 is interpreted as a 
measure of satisfaction with students ’ military environment; RC-3 is a measure 
of satisfaction with students ’ non-barracks/dormitory government quarters; RC-5 
is a measure of students ’ satisfaction with their service unit’s computer learning 
center. In view of the above RC-2, RC-3 and RC-5 are considered with only 
military personnel; conversely RC-1 and RC-4 are strictly academically oriented 
and include the GS data 

The non-parametric tests for central tendency with respect to military rank 
indicate that only RC-2 means differ significantly across among ranks. With 
respect to heteroscedasticity, RC-2 is the only factor that indicates significant 
differences in variances across treatment levels and the Welch Anova result of 
< 0001 still confirms the difference in central tendency. As previously stated, 
results for RC-2 are computed without the GS data but with the zero values in the 
data. To further investigate the nature of the dissimilarities among means for 
military rank two more variations are examined and they are as follows: (1) 
military rank without the zero values, (2) military rank without zeros or levels with 
fewer than ten observations. With these modifications, the test results for RC-2 
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continue to indicate that the null hypothesis should still be rejected; therefore, it is 
concluded that significant differences in satisfaction levels exist for various ranks. 
A graphical display of the measures of central tendency for rank versus RC-2 with 
all ranks and zeros included is presented below in Figure 9. 




Figure 9: RC-2 versus Military Rank 



At first glance it appears that there exist two general “levels” for the 
means: (1) enlisted mean at about 5.5 and (2) officer mean at about three. 
However, this is not necessarily the case. It is noted here that pay grades E8 and 
01 have only one observation each and that 02, 04, 05, and 06 all have fewer 
than ten observations each. When the Tukey Kramer Honest Significant 
Difference (HSD) test is employed we find that there exist similarities from officer 
to enlisted and differences within the enlisted group. The interested reader is 
referred to Appendix A, the matrix of HSD’ s where positive values indicate pairs 
of means that are classified as significantly different by that test. 
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D. EFFECTS OF SEX ON STUDENT SATISFACTION LEVELS 



In this case the treatment, sex, has two nominal levels, male and female. It 
appears that neither males nor females evidence a higher level of satisfaction for 
any of the factors - there are no significant results from the non-parametric tests 
for central tendency. 
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rV. CONCLUSIONS/RECOMMENDATIONS 



This chapter provides conclusions concerning the results from the various 
analyses performed in this thesis. Recommendations for possible use of the factor 
scores and suggestions for further study are also provided. 

A. PRINCIPAL COMPONENT ANALYSIS 

Although the principal components found here are interpretable, it is not 
recommended that their interpretations be used to describe the dimensions in the 
data. The principal components are useful for selecting initial factors for rotation; 
however, in this case, the rotated components provide a more thorough and 
intuitively defensible interpretation. 

B. FACTOR ANALYSIS 

The factor analysis performed here is deemed successful, and the author 
has a high degree of confidence in the interpretations provided for the five rotated 
components chosen. It is recommended that the two dimensional ASQ 2.0 
summary report currently used at DLI be modified to include the additional 
dimensions described in Chapter II of this thesis. The variables that load on RC-3, 
RC-4, and RC-5 are better correlated with those factors than with the two factors 
from DLI’s original design (academic and military). 

It is recommended that a similar analysis be performed on the ASQ 1.0 
data. If the same factors are determined to exist in that larger data set, the issue of 
trends across the sub-scales (sex, pay grade, and DLPT) can be more thoroughly 
explored, due to the larger number of observations that exist for ASQ 1.0. 
Additionally, this would allow the possible combination of the data bases, or at 
least a portion of them, yielding larger numbers of observations for any future 
studies of this nature. 

It is further recommended that DLI consider modifying its ASQ response 
set to include a “central scale” variable such as “neither agree nor disagree.” This 
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may help eliminate some of the “0” responses that may occur because respondents 
falsely believe that checking “no opinion” is a center scale response akin to the 
suggested “neither agree nor disagree.” 



C. FACTOR CENTRAL TENDENCIES 

The analysis in this section represents a good start toward a thorough 
investigation into trends in student satisfaction based on other population 
characteristics. Obviously sex, pay grade, and DLPT are not the only sub-scales 
worthy of consideration. It is recommended that further analysis be done in this 
area. Other areas of possible interest are the different schools/departments, or 
possibly students’ services. Exploration of a similar nature into these and other 
areas will allow DLI to further their knowledge of their student population, better 
equipping them to succeed in their goal of continuous quality improvement. 
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05 -3 46092 -0 05390 0 47062 0 19128 -0 48284 -0 70553 -142082 -1 51580 -3 81427 -4 33850 66927 -6 99498 -3 78317 -4 88712 -5 06011 

W1 -3 49576 -0 29612 0 16735 -0 11163 -0 79042 -1 01541 -1 72181 -1 80652 -3 98726 -4 48439 -96802 -7 02982 4 01129 -5 06011 -5 64316 

Positive values show pairs of means that are significantly different 
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Means Comparisons j 
Dif=Mean[i)-Mean[j] 
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